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A SIMPLIFIED EXAMPLE 
OF 

HOW TO EXTRACT MORE PLANNING DATA 
FROM 

EXISTING EVALUATION INSTRUMENTS 



Wouldn't it be nice to. have an empirically 
sound yet procedurally simple way to extract 
more planning data from existing evaluation 
instruments? 

Wouldn't it be nice to have a simple 
statistically sound way to identify the 
subgroups into which evaluating participants 
divide themselves? 

Wouldn't it be nice to have the simple 
uncomplicated devise that scientifically 
pinpoints both AGREEMENT and DISAGREEMENT among 
major evaluator subgroups, the director of the 
program being evaluated and the program staff? 



You guessed it, such material is available. 

From a practical point of view, the publication being talked about 
presents a sample evaluation step-by-step from data to findings, from 
findings to conclusions, and from conclusions to planning. 

Since every step of the way is both illustrated with examples and 
explained by careful analysis, this publication could be called a 
complete HOW-TO-REPLICATE-IT kit. 

This publication starts with the obvious fact that the questionnaire 

used last year, looked at and revised slightly by you this year, and 

duplicated for all evaluators probably doesn't have the following 

characteristics : 

Accurate enough to provide valid data. 
Short enough to be filled out by a large 
^ group of evaluators in a hurry to go home. 
Relevant enough to elicit concrete comments* 
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If you don't have an evaluation instrument with the above three 
characteristics, this document shows you how to develop one without having 
to reinvent the wheel or invest several days in the process. 

If the evaluation you used last year and revised this year has the 
above three characteristics ^ this publication shows you how to tabulate 
the evaluation accurately, rapidly, and systematically. 

Any reader who can do all of the above should stop reading this 
short article and write to the author immediately. Send the author a 
copy of what you have written. The author will be happy to incorporate 
your discoveries and insights into the next version of this article. 

No claim is made that this publication shows the one and only "best" 
method to evaluate. However, it is a different method. Just as teachers 
tend to teach as they were taught, so evaluators tend to imitate an 
evaluation that seems to have done a good job elsewhere in similar 
circumstances . 

This publication proposes one example of one evaluation type that 
seems to have worked in a specific set of circumstances. 



7 



THE AUDIENCE FOR THIS DOCUMENT 



This document is addressed to any educator who finds himself in the 
position of evaluating an educational gathering of adults. 

Typically, the evaluator in this case develops an evaluation form, 

explains and distributes the evaluation form, and adds up the total 

o 

number of responses. 

In the case where room has been left for participant reactions, the 
evaluator often types up the participant responses question by question. 

Such reports are usually accurate, even though much reading time is 
consumed. 



.Wouldn't it be nice to have a simple 
way to identify the subgroups into which 
participants divide themselves? 

Wouldn't it be nice to have a simple 
uncomplicated statistical device that 
scientifically pinpoints both AGREEMENT and 
DISA^EEMENT among these subgroups of participants? 

Wouldn't it be nice to have an example of 
how this has been done so that local implication 
would not be left to chance? 

Wouldn't it be nice to look at an evaluation 
instrument that provides not only a total of 
responses and a transcript of comments BUT ALSO 

Statistical findings. 

Conclusions based upon empirical evidence, and 
Planning based upon the reinforcement of 

previous success and the avoidance of 

past errors. 
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This document has something for any reader who answers yes to the 
above questions. It presents a sample evaluation step'-by-step from data 
to findings, from findings to conclusions, and from conclusions to planning. 

After the reader has had a chance to look at an actual evaluation 
case study, the reader has the option of stopping or of going on to the 
next section which composes a complete HOW-TO-REPLICATE-IT kit. 
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Do you find it difficult to develop a 
questionnaire with the following characteristics? 

• Accurate enough to provide valid data 

• Short enough to be filled out by a large group 
of evaluators in a hurry to go home 

• Relevant enough to elicit concrete comments 

After you have invested a lot of time in 
developing an adequate evaluation instrument 
with the above characteristics, are you able 
to tabulate it accurately, rapidly, and 
systematically? 

Anyone involved with evaluating an institution, workshop, conference, 
training session, or meeting is concerned with the above questions. 
Similarly, most evaluators are concerned with developing an appropriate 
evaluation instrument and with pinpointing the findings and conclusions 
available therefrom. 

The typical solution to this double challenge is to find out what 
was done last year and to repeat it with a few minor surface modifications. 
If last year's evaluation was adequate, then there is reason to hope that 
this year's replication will be at least adequate or better than last 
year • 

On the other hand, if a poor job was done last year, there is little 
reason to hope that a few surface modifications can improve this basic 
lack of quality evaluation. 
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Just as teachers tend to teach as they were taught, so evaluators 
tend to imitate an evaluation that seems to have done a good Job elsewhere 
in similar circumstances. 

This document is presented in two parts. Part I shows the evaluation 
instrument, the statistical treatment, the findings, the conclusions, and 
the plans developed as a result of evaluating one institute. Part II 
analyzes this example in a how-to-do-it kit that can be imitated in a 
variety of circumstances. 

Any reader who cannot yet decide whether or not this docxmient will 
be profitable for local applications is urged to do the following: 

First, read Part I to find out if this evaluation 
is similar and appropriate to local evaluation 
needs. 

Second, if this approach to evaluation is 
considered relevant, read Part II on how- to-do-it . 



PART I 

AN EVALUATION SUMMARY 
AND 

A COMPLETE EVALUATION REPORT 

An evaluation summary should be short and to the point. An evaluation 
summary is meant to be read rapidly and as such should concentrate only on 
the highlights. An evaluation summary should include specific sources of 
additional details for those interested. 

A complete evaluation report is intended to give the complete picture 
of an evaluation. A complete evaluation report is for thoughtful reading 
and point by point discussion. A complete evaluation report should indicate 
the exact statistical data sources upon which findings, conclusions, and 
planning recommendations are based. 

As herein defined, AN EVALUATION SUMMARY includes: 

Conclusions 
Rec ommend a t ions 

Cross-References for more detail 

As herein defined, A COMPLETE EVALUATION REPORT includes: 

The evaluation item text 

Evaluator response alternatives 

Findings 

Data source 

Conclusions 

Planning recommendations 
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For an example of a concise one-page evaluation summary, look carefully 
at page 

For an example of a typical complete evaluation report, start reading on 
page and continue until the end of part !• 

After this introduction to the same data presented in two different 
formats, that is, EVALUATION SUMMARY FORMAT and COMPLETE EVALUATION REPORT 
F(SIMAT, the reader is in a position to contrast specific characteristics and 
functions for each format type. 

For those interested in a step-by-step explanation of how to develop 
either an evaluation summary or a complete evaluation report, part II is 
provided. Part II begins on page 31., 
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EVALUATION CONCLUSIONS 
AND 

RECOMMENDATION 
AT A GLANCE 



V^^LNV^LiUD A.\JViD 




For More, 
jjeuaxi oee 


!• Rapport was achieved. 


1. A working relationship still, has to. 
be spelled out in detail. 


Pages 


2« Modules and Mastery Tests were 
produced. 


2. Standards are needed for: 
. . a) Quality control and ** 
b) Use with learners. 


Pages 


3. Some USABLE materials were produced. 


3. Tl:^se usable materialsneed to be 
catalogued and made available for 
general use. 


Pages 


4. Some public school educators became 
acquainted with some correctional 
institution problems. 


4. An exhaustive and authenticated list 
of correctional problems needs to be 
drawn up and discussed- 


Pages 


5. Some correctional institution 
educators became acquainted with 
some requirements of occupational 
subiect matter. 


5. A concise list of measurable 

requirements of occupational subject 
matter needs to be drawn up and 
discussed . 


Pages 


6. Most participants realized 
interagency cooperation poses 
protocol and procedural problems • 


6. Interagency protocol problems 
need to be spelled out in terms 
of "procedures to follow*" 


Pages 


7. Module expertise ranged from 
none to extensive . 


7. Sample existing modules should 
have been distributed at the very 
beginning of the institute and then 
adapted for local use- 


Pages 


8. The modular approach was found 
helpful "in general." 


8. A more systematic approach is needed 
to set standards; for example, 
levels of excellence in using 
modules: 

Level I: Modules are written. 
Level II: Modules are debugged 

in small groups . 
Level III: Modules are field 

tested with large groups . 
Level IV: Modules are used to 

produce documented 

measurable learner 

success • 


Pages 


9. The evaluator received three 
different agendas for the same 
institute . 


9. An agenda: 

a) Must be established in advance * 

b) Must be acceptable to all 
concerned . 

c) Must be adhered to and followed • 
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evaluator identification 



EVALUATION ITEM TEXT: 



CHECK 
ONE: 



I work for a 



□ public school 

□ correctional institution 



EVALUATOR RESPONSE ALTERNATIVES : 
ALTERNATIVE A: Check neither box. 
ALTERNATIVE B: Check the public school box 
ALTERNATIVE C: Check the correctional institution box 

FINDINGS : 

16 evaluators checked neither box. 

These evaluators constitute the NO IDENTIFICATION group, 

11 evaluators checked the PUBLIC SCHOOL box. 

These evaluators constitute the PUBLIC SCHOOL group, 

25 evaluators checked the CORRECTIONAL INSTITUTION boxT 

These evaluators constitute the CORRECTIONAL INSTITUTION group. 

DATA SOURCE : Page 45, . table 1, question 0, 

Obviously, the simple directions, "CHECK ONE," were not enough to 
get the NO IDENTIFICATION groups to check the appropriate box. 

Internal evidence indicates that at least 4 of the NO IDENTIFICATION 
group worked for a correctional institution. 

External evidence also raises the possibility that members of the 
NO IDENTIFICATION group might work for State Education Departments, State 
agencies, or other institutions. 

PLANNING RECOMMENDATIONS ; 

The following sentence should be added to the general directions: 
"Identify yourself as working for a public school, correctional institution, 
or other." 
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The evaluator identification question should be expanded to include, 



"Other; please specify. 
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EVALUATION ITEM 1 



EVALUATION ITEM TEXT ; 

Do you feel that this meeting between public school teachers and 
correctional teachers has produced a working relationship? 

EVALUATOR RESPONSE ALTERNATIVES ; 
ALTERNATIVE A; Check the box for yes, 
ALTERNATIVE B; Check the box for no. 

ALTERNATIVE C; Add comments in the section entitled HOW. 
FINDINGS : 

100% of all three groups (no identification, public school, 
correctional institution) responded to evaluation item with 
YES. 

52 evaluators provided a total of 45 comments in the section 
labeled HOW. 

DATA SOURCE ; Page 45, table 1, TOTAL COUNTS, question 1. 
Page 54 J COMMENTS ON ITEM 1. 

CONCLUSIONS ; 

It is rare to find such unanimity in a group that brings together 
two different types of educators from two different types of institutions. 

It can be assumed that in the opinion of evaluators, a working 
relationship has been developed. 

Operationally, this working relationship can be defined as an 
"exchange of ideas and interaction," according to more than 30 spontaneous 
evaluator comments. 



ERIC 



16 



PLANNING RECOMMENDATIONS: 



Operationally, a working relationship has been defined as "exchanging 
ideas and interaction." 

If the planners of the next institute have other important components 
to be included in a working relationship, the agenda, format, and evaluation 
instrument should reflect this concern for additional components, for 
example, such as 



Exchange of addresses 

Setting up of meetings 

Selection of joint curriculum areas 

A catalog of available testing materials 

Identification of priority problems 

Development of a timetable for action 

A chance to make definite time and personnel commitments 



EVALUATION ITEM 2 
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EVALUATION ITEM TEXT ; 

Was a module or mastery test 
completed as a result of this 
Institute? 



p Yes — >Which? 

□ Module 

□ Mastery test 
QNo )Why not? 



EVALUATOR RESPONSE ALTERNATIVES ; 
ALTERNATIVE A; Check the box for yes. 

ALTERNATIVE B; Identify whether a module was developed. 
ALTERNATIVE C; Identify whether a mastery test was developed. 
ALTERNATIVE D; Check the box for no. 

ALTERNATIVE E; Explain why neither a module or a mastery test 
was developed. 



FINDINGS ; 

100% of the evaluators in all three groups (no identification, 
public school, correction institution) checked the box marked 
yes. 

DATA SOURCE ; Page45, table 1, TOTAL COUNT, question 2, 

Not only was a working relationship developed (as established in 

question 1), but this institute resulted in the production of both 

modules and mastery tests. 

This means that the evaluation of the institute has gone from level 1 

to level 2. 

Level I refers to participant satisfaction. 
Level II refers to participant production of usable products, 
namely, modules and mas tery tests . 
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PLANNING RECOMMENDATIONS ; 

Usable products were developed at this institute. 

However, the institute evaluation makes no mention of how these 
products will be used on level III or level IV. 

Level III refers to field testing and modification of 

strategies, products, and materials. 
Level IV refers to documenting the benefits accruing 

to classroom teachers taught by participants 

of the institute. 

In addition, it may be useful to have a catalog of the modules and 
mastery tests developed. 

Since obviously an institute of this nature can permit each 
participant to produce either a module or mastery test, the quantity 
level of excellence must be set even higher. In order to come up with 
an appropriate measure of productivity, it might be advisable to ask 
how many modules and how many mastery tests were produced. 
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DETAILED ANALYSIS 0? EVALUATION ITEM 2 



M 

s i H 



O M 
O OH 



^ P 4 W M 



H h3 H 

o S5 



BOTH 29 8 7 14 

MODULE 18 6 4 8 

MASTERY TEST 5 2 0 3 

NEITHER 0 0 0 0 



The above detailed analysis of evaluation item 2 indicates that 

29 evaluators developed both modules 
and mastery tests. 

18 evaluators developed 'modules . 

5 evaluators developed a mastery test. 

In general, a module is composed of 4 components: 

OBJECTIVES 
PREASSESSMENT 

LEARNING ACTIVITIES AND ENVIRONMENTS 
MASTERY POSTTESTS 

If this definition of a module was followed, it would seem that the 
vast majority of the participants have achieved success. 

On the other hand, the way the way the question is answered, it 
might be that the participants perceived modules and mastery tests as 
two different elements Instead of conceiving a mastery test as the 
verification of the success of a given module. 
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EVALUATION ITEM 3 

EVALUATION ITEM TEXT ; 

Do you feel that what you produced will be useful to you in your 
teaching situation? 

EVALUATOR RESPONSE ALTERNATIVES : 

ALTERNATIVE A: Circle yes, 

ALTERNATIVE B: Circle 100%. 

ALTERNATIVE C: Circle 75%, 

ALTERNATIVE D: Circle 50%,. 

ALTERNATIVE E: Circle 25%- 

ALTERNATIVE F: Circle No-0%, 

ALTERNATIVE G: Do not circle anything, 

ALTERNATIVE H: Comment in response to HOW, 

FINDINGS : 

1007o success or better was claimed by: 

25% of all three groups of evaluators, 

27% of public school evaluators, 

36% of correctional institution evaluators. 

75% success or better was claimed by: 

73% of the GRAND TOTAL of all evaluators. 

82% of public school evaluators, 

75% of correctional i.ustituti6n evaluators, 

50% success or better was claimed by: 

94% of GRAND TOTAL of all evaluators, 

91% of public school evaluators. 

96% of correctional institution evaluators. 
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Therefore, it is proposed that at the next meeting: 

A. Modules be defined in writing 

B. Mastery tests be defined in writing 

C. A list of criteria be given in writing that 
clearly identify acceptable completion of 
both a module and mastery test 



t 
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DATA SOURCE : 

Page 48, table 4, TOTAL COUNT AND SUBTOTALS CONVERTED TO CUMULATIVE 
.PERCENTAGES, question 3, 

Refer especially to findings labeled 3A, SB, and 3C. 
CONCLUSIONS ; 

More than 90% of the group felt that AT A BARE MINIMUM at least half 
or more of the material produced will be useful in actual teaching situati 

It might be observed that the comments given in answer to evaluation 
item 3 provide concrete examples of level II application of teacher 
institutes to classroom teaching. 

PLANNING RECOMMENDATIONS : 

So far, the first three questions have produced such an overwhelming 
positive response that the evaluator must become suspicious of his tactics 
used for evaluation. 

Perhaps, the questions were of such a nature that only a positive 
response could be given. 

Perhaps, the level of quality and expertise expected was set too 
low when consideration is made of the high caliber of participants in 
attendance. 

Similarly, it can be an eye-opener for teachers "to find out about 
the vastly different types of applications made by similar educators In 
a different environmental situation. 
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EVALUATION ITEM 4 



EVALUATION ITEM TEXT ; 

Did this institute acquaint the public school teachers with the 
problems and needs of the correctional institutions? 



EVALUATOR RESPONSE ALTERNATIVES: 


ALTERNATIVE 


A: 


Circle 


yes 


ALTERNATIVE 


B: 


Circle 


100% 


ALTERNATIVE 


C: 


Circle 


75% 


ALTERNATIVE 


D: 


Circle 


50% 


ALTERNATIVE 


E: 


Circle 


25% 


ALTERNATIVE 


F: 


Circle 


No-0% 


ALTERNATIVE 


G: 


Comment in the s 



FINDINGS : 
1007. Success is claimed by: 

45% of the public school evaluator3. 

40% of the correctional institution evaluators. 

75% or more success is claimed by: 

91% of the public school evaluators. 

52% of the correctional institution evaluators. 

This means that 39% of the public school evaluators 
held a higher, opinion than correctional institution 
evaluators. 

This difference is found to be statisticdly significant 
on the I7a level of confidence (P .01.) 

DATA SOURCE : 

page 48, table 4, TOTAL COUNT AND SUBTOTALS CONVERTED TO 
CUMULATIVE PERCENTAGES, question 4, finding 4A and 
finding 4B. 

Page 30, DISPLAY 2, TABLUATING TABLE FOR SIGNIFICANT TESTING 
(Slide rule accuracy), section 4b. 
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CONCLUSIONS; 



91% of the public school evaluators feel that the public school 
teachers have done a good job in becoming acquainted with the 
problems and needs of correctional institutions • 

This opinion is shared by only 527. of the correctional 
institution evaluators. 

In general, the comments coming from public school evaluators 
differ from the comments coming from correctional institution 
evaluators in the sense that there are few overlapping comments. 



PLANNING RECOMMENDATIONS ; 

In order to simplify this process of becoming aware of the problems 
and needs of correctional institutions, a master list should be 
developed pinpointing correctional problems. 

This master list with correctional problems and needs might include, 
but not be' limited to? 

Types of teaching manpower available 
Levels of learning 
, Discipline 

Security procedures 
Placement 

Quorum difficulties 
Job placement 
Administration problems 

Since many of the public school teachers presumed that correctional 

institution problems were similar to the teaching problems encountered 

in occupational centers, it might be good to prepare a specific list 

that would point up the contrasts. 
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EVALUATION ITEM 5 




* 


EVALUATION ITEM TEXT: 






Did this workshop acquaint the correctional teachers more 






fully with the requirements of occupational subject matter? 


• 




EVALUATOR RESPONSE ALTERNATIVES: 






ALTERNATIVE A: Circle yes 






ALTERNATIVE B: Circle 1007. 






ALTERNATIVE C: Circle 75% 






ALTERNATIVE D: Circle 50% 






ALTERNATIVE E: Circle 25% 






ALTERNATIVE F: Circle No-0% 






ALTERNATIVE G: Comment in the section marked IN WHAT WAY. 






FINDINGS: 






100% success was claimed by: 






27% of public school evaluators 


- 




32% ,of correctional institution evaluators 






75% or more success was claimed by: 






36% of public school evaluators 






64% of correctional institution evaluators 






(The level of confidence here is 11%.) 






(P^.ll) 






50% success or more is claimed by: 






45% of public school evaluators 






88% of correctional institution evaluators 






(The level of confidence here is 17#.) 






(P<;;;^ .01) 
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DATA SOURCE : 

Page 48, table 4, TOTAL COUNT Ar© SUBTOTALS CONVERTED TO CUMULATIVE 
PERCENTAGES, question 5, finding 5A, finding 5B, finding 5C, 

page 50, DISPUY 2, TABULATING TABLE FOR SIGNIFICANCE TESTING • 
(Slide rule accuracy), section 5B, section 5C. 

CONCLUSIONS ; 

88% of the correctional institution evaluators feel that correctional 
institution teachers have done at least an average job of becoming 
acquainted more fully with the requirements of occupational subject/ 
matter. 

This opinion is shared by only 45% of the public school evaluators. 

Allowing for the relative factor that each group will consider 
itself better than the other group, it still remains that statistically 
significant differences on the same instrument poJ,nt up the fact that 
the public school teachers do not feel that the correctional institution 
teachers have become sufficiently acquainted with the requirements of 
occupational subject matter. 

PLANNING RECOMMENDATIONS ; 

During the 3 day institute, it was felt that casual interaction would 
provide sufficient opportunities for correctional teachers to become more 
fully acquainted with the requirements of occupational subject matter. 

According to the evaluation instrument, this familiarity was neither 
claimed by correctional institution teachers in large numbers nor backed 
up by the corresponding opinion of public school teachers. 

However, both groups seem to feel that at least an average (50%) 
type of success was achieved by the interaction. 

The comments of public school teachers to question 5 were only 4 in 
number. This does not give much of an indication of what the correctional 
institution teachers still have to do. 

The comments of correctional institution teachers number 15 and stress 
"B^Wi^ix-^hlTi^-^^^ - 
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It might be good for public school teachers to look at the following 
list find make any additions or suggestions necessary to pinpoint the 
"requirements of occupational subject matter." 

The following occupational subject matter requirements were listed 
by correctional institution teachers: 



Paperwork 
Student levels 
Uniform curriculum 
Performance objectives 
Learning mentality 
Materials 

Specific techniques 
Interaction 
Modernization 
Positive approach 
Sources of materials 



0 
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EVALUATION ITEM 6 



EVALUATION ITEM TEXT ; 

Did this workshop acquaint both' the public school and the correctional 
institution teachers with the problems encountered in interagency 
cooperation? 



EVALUATOR RESPONSE ALTERNATIVES ; 
ALTERNATIVE A: Circle yes 
ALTERNATIVE B; Circle 100% 
ALTERNATIVE C; Circle 75% 
ALTERNATIVE D; Circle 50% 
ALTERNATIVE E; Circle 25% 
ALTERNATIVE F: Circle No-0% 

ALTERNATIVE G; Comment in the section marked WHICH PROBLEMS. 
FINDINGS ; 

100% success was claimed ^by: 

36% of the public school evaluators 

32% of the correctional institution evaluators 

75% or more success was claimed by:. ^ 

91% of the public school evaluators 

64% of the correctional institution evaluators 

(*rhe level of confidence is 5%) 
(P<^.05) 

50% or more success was claimed by: 

91% of the public school evaluators 

80% of the correctional institution evaluators 
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DATA SOURCE: 



Page 48, table 4, TOTAL COUNT AND SUBTOTALS CONVERTED TO CUMULATIVE 
PERCENTAGES, question 6, finding 6A, finding 6B, finding 6C* 

Page 50, DISPLAY 2, TABULATING TABLE FOR SIGNIFICANT TESTING (Slide 
rule accuracy) section 6B. 



CONCLUSIONS : " 

At least average success in acquainting both the public school 
teachers and the correctional institution teachers with some 
of the problems encountered in interagency cooperation has been 
claimed by: 

91% of public school evaluators 

80% of correctional' institute evaluators 

A better than average success has been claimed by: 

91% of public school evaluators 

64% of correctional institution evaluators 

In the perspective of question 4 and question 5, the results of 
question 6 can be interpreted to mean that even though technical 
correctional institution requirements and technical occupational 
subject matter requirements have not been completely mastered, 
the two groups have got together and untangled some of the 
problems that arise whenever interagency cooperation occurs. 



PLANNING RECOMMENDATIONS; 



Specific problems encountered in interagency cooperation were enumerated 
as follows: 

Bureaucratic protocol 
Eliminating red tape 
Preplanning visits 
Security requirements 
Lack of standardization 
Difference in student types 

If other problems are envisioned by the planners of the institution, it 

might be a good idea to prespecify these problems and to direct the responses 

to pinpoint priority problems. 
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EVALUATION ITEM 7 



EVALUATION ITEM TEXT ; 

Before this workshop have you ever been involved in any 
way with occupational modules? 

EVALUATOR RESPONSE ALTERNATIVES ; 

ALTERNATIVE A: Check the box for yes. 

ALTERNATIVE B: Check the box for no. 

ALTERNATIVE C: Comment in the section marked SPECIFY. 

FINDINGS : 

73% of the public school evaluators checked the box marked yes. 

36% of the correctional institution evaluators checked the box 
marked yes . 

(The level of confidence here is 5%) 
(P <.05) 

(P >.oi) 

DATA SOURCE : 

Page 48, table 4, TOTAL COUNT AND SUBTOTALS CONVERTED TO 
CUMULATIVE PERCENTAGES, question 6, finding 7. 

CONCLUSIONS ; 

Both public school evaluators and correctional institution evaluators 
claim familiarity with modules. 

The statistical difference would indicate that more public school 
teachers than correctional institution teacher are familiar with modules. 

Looking at the comments to question 7 reveals that modules have been 

studied in a number of different environments: 

Inservice education 
Teacher training 

pj.^^g^g iriclustry " ^ ~ ~ 

Personal experience - 
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PLANNING RECOMMENDATIONS ; 

Question 7 has elicited from the participants how each participant 
was previously exposed to modules. 

It might be worthwhile during a future institute to find out how 
each participant actually USES modules in teaching, testing, and programming. 

The professional evaluator must develop a tendency to mistrust data 
that is supported only by opinion and not by hard product. 

This means that it strikes the present evaluator as unusual that so 
many claims were made for modules without several modules being actually 
presented as evidence. 

At the next institute, it might be a good idea to request participants to 

the following for e^ch module they have already produced. 

Name of module 
Name of author 
Address of author 
Procedures to obtain a copy 

This kind of information would provide a baseline of what already 
exists. Such a baseline would prevent needless duplication of existing 
materials. 



5^ 

- w^«s5 — 
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EVALUATION ITEM 8 



EVALUATION ITEM TEXT : 

Do you feel that this institute aided you in the development 
of a modular approach for your course of study? 

EVALUATOR RESPONSE ALTERNATIVES; 



ALTERNATIVE 


A: 


Circle 


yes 


ALTERNATIVE 


B: 


Circle 


100% 


ALTERNATIVE 


C: 


Circle 


757. 


ALTERNATIVE 


D: 


Circle 


50% 


ALTERNATIVE 


E: 


Circle 


25% 


ALTERNATIVE 


F: 


Circle 


No-0% 


ALTERNATIVE 


G: 


Comment 


: in the section marked HOW 



FINDINGS : 

75% success was claimed by: 

46% of public school evaluators. 

64% of correctional institution evaluators. 

50% or more success was claimed by; 

92% of public school evaluators. 

80% of correctional institution evaluators. 

DATA SOURCE: 

page 48, table 4, TOTAL COUNT AND SUBTOTALS CONVERTED TO 
CUMULATIVE PERCENTAGES, question 8, finding 8A, finding 8B. 

CONCLUSIONS : 

Approximately 80% of all participants feel that the institute was at 
50% successful. 
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The comments on question 8 give specific examples of why this 
institute was considered successful. 

In these comments, it is to be noted that certain participants 
started at zero as far as awareness of modules was concerned. 

PLANNING RECOMMENDATIONS : 

As in evaluation item 2, it is recommended that participants be 
-exposed to at least 4 different levels of the modular approach: 

Level I: Participants develop modules composed of 

Objectives 
Freassessment 
Learning activities 
Mastery posttest 

Level II: Participants use modules with small groups 
of learners in order to debug the modules. 

Level III: Participants field test modules with large 
groups of learners. 

Level IV: Participants document the effectiveness of 
modules in producing measurable learner 
success • 

In this way, beginners can concentrate on Level I while those who 
have already mastered Levels I and II can concentrate on Levels III and IV. 

This will enable each participant to specify the number of levels 
of progress that have been achieved during a given session. 

There is always something for a teacher to learn new about modules. 
Unless a challenge is provided for the advanced and unless remediation is 
provided for those unfamiliar, such a workshop can be at cross purposes as 
the speakers try to please both those possessed of advanced information about 
modules and those possessed of no information about modules. 
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WHICH AGENDA? 
WHICH AGENDA?? 
WHICH AGENDA??? 



As a footnote to the above, it might be observed that the evaluator 
was presented with three different agendas for the same Institute. 

Obviously, one agenda was prepared several weeks before the 
Institute. Similarly, the second agenda was probably prepared a week 
or so before the institute. This means that the final agenda may have 
been confirmed as late as the night before the Institute. 

These changes were most likely motivated by sudden Insights and 
last minute planning. However, such sudden insights and last minute 
planning have a dlsasterous effect on both staff and participants who 
come expecting one thing and find another. 

In order to succeed, an agenda 

A. Must be established in advance, 
" B. Must be acceptable to all concerned, and 
, C« Must be adhered to and followed. 

If a meeting starts off with the anticipated agenda being annuled 
as the first activity on the docket, participants can be expected to 
manifest confusion, uncertainity, and insecurity. 

It is much more reassuring to arrive at a meeting knowing what to 
expect and to witness expectations becoming reality. 



PART II 



HOW TO DEVELOP 
EVALUATION SUMMARIES 
AND 

COMPLETE EVALUATION REPORTS 

Readers have had a chance to exam both the EVALUATION SUMMARY FORMAT and 
the COMPLETE EVALUATION REP(XIT FORMAT in part I. 

From a practical point of view, the evaluation summary format is the 
typical end product of most evaluations. After all, the evaluation summary 
format summarizes all vital information in the form of conclusions .and 
recommendations. It is not unusual for an evaluation summary to be rounded 
out with more details placed in an appendix. The typical appendix contains 
forms, displays, data tables, and evaluator comments. 

From a developmental point of view, the complete evaluation report format 
contains the 101 things that were spelled out in the mind of the overall 
evaluator who tabulated and consolidated the individual evaluation forms of 
many evaluators. 

As is obvious from a cursory glance of part I, every evaluator makes a 
selection from the total components of the complete evaluation report in 
order to fit the particular objectives of a specific evaluation. 

Except in rare exceptions, the typical complete evaluation report will 
be much shorter in length and words than the sample presented in part I. 
The sample in part I is extra long in order to make sure that a wide variety 
of alternatives is presented from which the evaluator may choose the most 
appropriate components. 
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Decisions require information. Evaluation summaries and complete 
evaluation reports are conceived as practical ways to provide information 
in a format that can lead to effective decisions. 
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MODIFY THE EXISTING FORM 
FOR COMPARABILITY 



There is no need to develop a special evaluation form to use with 
the methods herein explicated. 

However, it is necessary that the existing fom be modified, if 
necessary, for comparability. 

Comparability means being able to identify the TYPE OF PERSON who 
filled out a specific form. This is not the same as identifying the 
INDIVIDUAL who filled out a specific form. 

One simple way to do this is to provide a question on the form that 
identifies the type of respondent or the group in which the respondent 
is placed. 

PRACTICAL EXAMPLE 
Look at form A found on page 43 . 

This form is a modified version of a previously used form. 

The basic modification was to add a small section wherein each 
evaluator is identified as being either a public school educator or a 
correctional institution educator. 

There are two practical pieces of advice here: 

First, make sure that the group identification 
provided is sufficient to cover the entire range 
of possibilities. To make sure of this, it is 
sometimes advisable to add a group characterized 
as OTHER. 

Second, make sure that enough attention is given 
to this group identification so that each evaluator 
checks off the appropriate space. This additional 
stress avoids coming up with a large NO IDENTIFICATION 
group. 
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EVALUATE 
EACH EVALUATION ITEM 
FOR EACH EVALUATOR GROUP 



Some evaluation tabulators make the error of coming up with the 



total count as the first end product. 

This error is time consuming for two reasons: 

First, it is more accurate to add up each 
individual group and then doublechecked with 
the group total. 

Second, when the counts from two or three 
groups are merged into a total count, 
the total count often gives the impression 
of a total group concensus that is not 
present. 



PRACTICAL EXAMPLE 



Look at the following tabulation table. 

On the left hand side, the following is identified: 

Item number 
Response alternatives 
Space for yes 
Space for no 

Space for NR (No response) 



Across the top, the overall total and three group subtotals 
are identified: 

Total 

No Identification 
Public School 
Correctional Institution 
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Item 1 
Response Alternative 


Total 


No Identificatio 


Public School 


Correctional Ins 


Yes 










NO 










NR (No response) 











The fastest way to tabulate is to take the no identification group 
and divide the papers into three piles: yes, no, and NR. 

After each pile has been sorted and doublechecked, the count for 
each pile is put in the appropriate space. 

The same thing is done with the public school pile and the 
correctional institution pile. 

It is important that the right hand side of the tabulation page 
specify all groups being counted. 
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In proceeding to item 3, the response alternatives on the left hand 
side would include: • 



Yes 

1007. 

75% 

50% 

25% 

No-0% 

NR 



This would simply mean adding more lines to the left hand column 
of a tabulation page AND making sure that a different pile was used for 
each of the possible response alternatives. 

The time to check and recheck the count is before filling in the 
exact number on the tabulation page. 

After all the groups have been counted, the total score can be 
determined by addition and by double checking to avoid arithmetic 
errors. An empty sample tabulating table appears below: 



Item 3 
Response Alternative 


Total 


No 

Identification 


Public 
School 


Correctional 
Institution 


Yes 










100% 










757. 










50% 










25% 










No-0% 










NR 1 
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SUMMARIZE ALL TABULATION DATA 
IN . 
A ONE PAGE 
TOTAL COUNT AND SUBTOTAL 
TABLE 



Look immediately at table 2, TOTAL COUNT AND SUBTOTALS, found on 
page 46. 

This page represents the one page end product of the above procedure 
performed on the data used to illustrate this evaluation method. 
Note the following: 

A. The range of alternatives are placed on the left hand side. 

B. The groups are placed in order from left to right as follows: 

Totals 

No identification 
Public schools 
CorrectionaT institution 

C. The columns going from left to right are given a large amount 
of space because only 8 questions are summarized. If more 
than 8 questions were summarized, the page would be divided 
into two columns, 8 questions on the left, 8 questions on 
the right. 

D. The listing of alternatives in the left hand column form a 
Likert ranging from yes to 0 and no response. 

E. Because the three subgroups are unequal in number, it is 
very difficult to draw conclusions by looking at the 
numbers • 
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TRANSFORM TO PERCENTAGES 



TRANSFORM THE 
•^TOTAL COUNT AND SUBTOTALS" TABLE 
INTO A TABLE SPECIFYING 
'*T0TAL count and subtotals CONVERTED TO PERCENTAGES" 



Look at table 3, TOTAL COUNT AND SUBTOTALS CONVERTED TO PERCENTAGES 
found on page 47. 

This table represents the end product of the above procedure. 
Note the following: 

A. Since all respondents chose the same response for Items 1 
and 2, further statistical treatment of these items is 
unnecessary. Findings can be made immediately. 

B. Conversion to percentages enables each individual response 
alternative to be studied on a group by group basis. Placing 
a ruler under each line enables the reader to isolate the 
comparisons desired, 

C. The groupings have been placed more closely together on 
the left hand side in order to facilitate comparisons. 
As is evident, there is room for still other analyses 
on the same page. 

D. In a percentage table, special notice should be given 
to outstanding high percentages. For example: 

Look at 81% and 73% in item 7. 
Look at 50% and 55% in item 3. 
Look at 50% and 55% in item 5. 
Look at 55% in item 6. 
Look at 50% in item 8. 

Each of these noticably different percentages should be 
examined for possible clues leading to the identification 
of underlying trends in the data. 

i 

E. When one wishes to draw an overall conclusion about the 
trend of alternative responses, table 3 becomes inadequate. 
A cumulative percentage analysis is necessary to understand 
overall trends. 



USE PERCENTAGE TABLES WHILE 
TRANSFORMING TO PERCENTAGES 



We all know that ^ is 50% and that 1/4 is 25%. 

For these simple numbers, we don't have to use percentage tables. 

Whenever a conversion is made from a count to a percentage, it is 
advisable to use a percentage table. 

For examples of percentage tables, see display 3, display 4 and 
display 5 on pages 51, 52, and 53. 

These tables provide accuracy, consistency, and speed in the process 
of transforming numbers into percentages. 

In addition, tables of percentage equivalents can be used to go 
backwards from percentages to the number of individuals represented by 
a particular percentage. 

As a rule of thumb, percentage tables are accurate enough for most 
uses when not carried to decimal points. However, it must be noted that 
certain subtotals require rounding off in order to make sure that each 
subgroup has a total of 100 rather 99 or 101. Whenever this is done, 
make sure that the degree of accuracy as measured by the sample size is 
left intact. 
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CONVERT TO CUMULATIVE PERCENTAGES 



CONVERT THE 

"TOTAL COUNT AND SUBTOTALS CONVERTED TO PERCENTAGES" TABLE 

INTO A TABLE SPECIFYING 
"TOTAL COUNT AND SUBTOTAL CONVERTED TO CUMULATIVE PERCENTAGES" 



Look at table 4, TOTAL COUNT AND SUBTOTALS CONVERTED TO CUMULATIVE 
PERCENTAGES, found on page 43. 

Table 4 is an example of the end product of the above procedure. 
Note the following: 

A. Several findings have been pinpointed for each item. 

B. In general, the finding boiled down to pinpointing: 

Evaluator agreement 
Evaluator disagreement 

C. Evaluator agreement can be used to specify both concensus 
andtthe degree of concensus. 

D. Evaluator disagreement can be used to pinpoint divergent 
interpretations . 

E. Significant evaluator disagreements have been statistically 
tested in order to determine the level of confidence. 

F. The lower the level of confidence, the less chance of 
statistical error. 

G. It must be remembered that because of the varying sizes 
of each subgroup, no one rule can be laid down for how 
njany percentage points of difference constitute a 
significant difference. Each individual difference must 
be tested separately. 

TECHNICAL NOTE : 

Individuals who wish to understand the statistical test used to 
test the difference between percentages are referred to display 1, 
COMPUTING THE SIGNIFICANCE OF DIFFERENCE IN PERCENTAGES, found on page 



CATEGORIZE THE COMMENTS OF EVALUATQRS 
ITEM BY ITEM 



Look at page 54, COMMENTS ON ITEM 1, which does the following: 

A. All comments are reproduced 

B. All comments are separated into the three groups of: 

No identification 
Public School 
Correctional Insitutions 

C* A subtotal of comments is made for each subgroup 

D. A grand total of comments is made for all s,ubgroups 

Look at page 56, COMMENTS ON ITEM 3, 

In addition to the above functions, this listing of comments 
further subdivide the comments into two groups: 

I 

Group 1 reproduces the comments who 
evaluated item 3 75% or above. 

Group 2 summarizes the comments of 
those who evaluated item 3 50% or 
below. 

In the interpretation of item 3^ it is interesting to note that 
those individuals who made a positive evaluation followed up with a 
larger number of comments. 

Similarly, those indi3?^iduals who made a negative or lower evaluation 
Were unable to specify exactly the cause of the lower evaluation. 
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Those interested in actually going through the mathematical statistics 
used to ascertain the levef\of probability are referred to display 2, 
TABULATING TABLE FOR SIGNIFICANCE TESTING (SLIDE RULE ACCURACY) , found 
on page . 

It is to be noted that the placing of data in column fashion proceeding 
from top to bottom permits the statistician to double check visually the 
accuracy of all computations. 

It is noted that in most cases, slide rule accuracy is equivalent 
to the additional decimal places that could obtained easily be using an 
electronic calculator. 

The error curve for testing the difference between percentages with 
the sample size below 200 individuals is such that slide rule accuracy 
is just as discriminating as electronic calculator accuracy. 

In the case where the sample size would exceed 200, a much smaller 
percentage difference in responses would be found statistically 
significant. 

Two blank columns have been left on the right hand side of display 2 
in order to provide space for readers who wish to practice, utilize, or 
try out this test of statistical significance. The formula appearing 
in display 1 has been translated into the 18 sequencial steps found in 
display 2. 
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FORM A 
INSTITUTE EVALUATION 



43 



ERIC 



DIRECTIONS: Check the box for Yes or .No on questions 1, 2, and ?• For the other 
questions, circle the percentage from Yes (100%) to No (less than 25%) that 
best answers the yes-no type questions. When appropriate, add comments and 
explanation in the same space, 



1. Do you feel that this meeting between 
public school teachers and correction- 
al teachers has produced a working 
relationship? 



2. Was a MODULE or ^LA.S^ERY test completed 
as 'a result of this institute? 



5. 



ONE: 



I work f.or a 



□ public school 

□ correctional institution 



Do you feel that what ycu produced 
will be useful to you in your 
teaching situation"! 



Did this institute acquaint the public 
school teachers with the problems and 
needs of the correctional 



institutions'? 



1, /_/ Yes 
n No 



1. I / Yes. 



/ / No» 



How^ 



->Which? 

□ Module 

□ Mastery test 
Why not? 



3-. Yes- 100% 75% 50% 25%. No-0% 
How? 



4. Yes- 100% 75% 50% 25% No-0% 
Which ones especially? 



Did this v/orkshop acquaint the correc- 
tional teachers more fully with the 
requirements of occupational subject 
matter? 



Did this workshop acquaint both the 
public school teachers and the correc- 
tional teachers with some of the 
problems encountered in interagency 
cooperation? 



7. Bo,fore this workshop have you ever 

been involved in any way with occupa- 
tional modules? 



8. Do you feel that this institute aided 
you in the developmenc of a modular 
approach for your course of study? 



5. Yes- 100%' 75% 50% ' 25% N6-0% 
In what way? 



6. Yes- 100% 75% 50% 25% No-0% 
Which problems? 



7. I_l Yes 
n No 



Specify, 
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Flcasa add any other commencs abouc this 
institute:, or about this interagency 



8. Yes- 100% 75% 50% 25% No-0?! 
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Comments may bo written on the 
other side of this paee. 



FORM B 
TABULATION TABLE 



ITEM # 



o 
u 

00 



o < 



CO 1-^ 

AJ 

O CU 
•Q O 

0 u 

CO O 



CO Csl 
O O4 
•Q O 
CO O 



CO CO 
u 

o cu 

•Q O 

CO O 



CO ^ 
u 

o cu 

O 

3 u 
CO O 



CO in 
u 

o cu 

w J) 

M o 

:3 ^ 

CO o 



Response 

Alternative 

A: 



Response 

Alternative 

B: 



Response 

Alternative 

C: 



Response 

Alternative 

D: 



Response 

Alternative 

E: 



Response 

Alternative 

F: 



Response 

Alternative 

G: 
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TABLE 1 



TOTAL COUNT 
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ITEM RESPONSE GIVEN 

0) 16 = No Identification 
11 « Public School 

25 » Correctional Institution 

1) 52 » Yes 

0 = No 

0 = No Response 

2) . 52 = Yes 

0 = No 

0 = No Response 

3) 1 «= Yes 
13 = 100% 
24 = 75% 
11 = 50% 

0 = 25% 

1 = 0% 

2 = No Response 



4) 9 = Yes 

13 = 100% 

14 = 75% 
4 = 50% 
1 = 25% 
0 = 0% 

11 «= No Response 



5 




Yes 


10 




100% 


17 




75% 


9 




50% 


1 


at 


25% 


3 




0% 


7 




No Response 


9 




Yes 


7 


a 


100% 


21 




75% 


5 




50% 


3 




25% 


1 




0% 


6 




No Response 



7) 20 = Yes 
32 = No 
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8) 4 = Yes 

8 = 100% 
20 = 75% 
10 «= 50% 

8 = 25% 

1 = 0% 

1 = No Response 
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TABLE 2 



RESPONSE GIVEN 



ITEM 



TOTAL COUNT AND SUBTOTALS 



TOTAL 



S5 H 



CO 

§ 

u 



H 
H 

u 

o ss 



Yes 
No 

No Response 



1) 



52 
0 
0 



16 
0 
0 



11 
0 
0 



25 
0 
0 



Yes 
No 

No Response 



2) 



52 
0 
0 



16 
0 
0 



11 
0 
0 



25 
0 
0 



Yes 

100% 

757. 

50% 
25% 
0% 

No Response 



3) 



1 
13 
24 
11 
0 
1 
2 



0 
2 
8 
5 
0 
0 
1 



0 
3 
6 
1 
0 
1 
0 



1 
8 
10 
5 
0 
0 
1 



Yes 

100% 

75% 

50% 

25% 

0% 

No Response 



4) 



9 
13 
14 
4 
1 
0 
11 



2 
5 
6 
0 
0 
0 
3 



1 
4 
5 
1 
0 
0 
0 



6 
4 
3 
3 
1 
0 
8 



Yes 

100% 

75% 

50% 

25% 

0% 

No Response 



5) 



5 
10 
17 
9 
1 
3 
7 



2 
2 
8 
2 
0 
il 
1 



1 
2 
1 
1 
0 
0 
6 



2 
6 
8 
6 

1 
2 
0 



Yes 

100% 

75% 

50% 

25% 

0% 

No Response 



6) 



9 
7 

21 

5 
3 
1 
6 



3 
1 
7 
1 
0 
0 
4 



2 
2 
6 
0 
1 
0 
0 



4 
4 
8 
4 
2 
1 
2 



Yes 
No 



7) 



20 
32 



3 
13 



8 

3 



9 
16 



Yes 
100% 
75% 
50% 
O 25% 

ERiCo% 

™^™No Response 



8) 



4 
8 

20 
10 
S 

1 
1 



1 

2 
8 
2 

2 
0 
1 



1 
1 
3 
4 
1 
1 
0 



2 
5 
9 
4 
5 
0 
0 



TABLE 3 
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TOTAL COUNT AND SUBTOTALS 
CONVERTED TO 
PERCENTAGES 



(Rounding has been performed question 













by 

J 


RESPONSE 




< 

H 




9 ^ 




GIVEN ITEM 


O 
H 




» H 




Yes 


1) 


1007 




1007 




No 




0 




0 








0 




0 




Yes 


2) 


100 




100 




No 




0 




0 




No Response 








0 


• 

H 










C/j 








H 


jz; H 


Pi 




Yes 


3) 


2 


0 


0 




1 new 








27 


32 


757. 




46 


so 




40 


50% 




21 


31 

mJ X 


Q 


20 


257. 




0 


0 


0 


0 


07. 




2 


0 


9 


0 


No Response 




A 


o 


n 
u 


A 

H- 


Yes 


4) 


1 7 




Q 


OA 


lUU/. 








JO 


1 A 


757. 






7 


/i A 


1 0 


507. 




O 
O 


u 


Q 




257. 




2 


0 


0 


12 


07. 




0 


0 


0 


4 


No Response 




21 


19 

x^ 


0 


32 


Yes 


5) 




12 5 

X ^ • ^ 


9 


8 


100% 




19 


12 5 

x^ • ^ 


18 


24 


75% 




33 


50 


9 


32 


50% 




17 


13 


9 


24 


25% 




2 


0 


0 


4 


0% 




6 


6 


0 


8 


No Response 




13 


6 


55 


0 


Yes 


6) 


17 


19 


18 


16 


100% 


13 


A 
\j 


1 R 


16 


75% 












50% 




10 


6 


0 


16 


25% 




o 


U 


Q 


Q 
O 


0% 




2 


0 


0 


4 


No Response 




1 0 


0 


U 


o 
o 


Yes 


7) 


38 


19 


73 


36 


No 


62 


81 


27 


64 


Yes 


8) 


8 


5.5 


9 


8 


100% 




15 


13 


9 


20 


75% 




39 


50 


28 


36 






19 






xu 
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15 


0 


9 


20 






2 


0 


9 


0 



(i< CO 
100% 

0 

0 

100 
0 
0 



PS CO 
o » 

O M 

100% 
0 
0 



100 
0 
0 



Finding 



Finding 
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TABLE 4 



RESPONSE 
GIVEN 
Yes 
100% 
7 75% 
' 50% 
' 25% 
0% 



ITEM 
3) 



H 
2 
27 
73 
94 

94 
96 



§ 

H 
H 

H 



e 

o 

:a 

0 
13 
63 
94 
94 



CO 

a 

H 

(Li 
0 

27- 
82- 
91- 
91 



94 100 



No Response 100 100 100 100 



g TOTAL COUNT AND SUBTOTALS 

H CONVERTED TO 

w CUMULATIVE PERCENTAGES 

g 

4 

.36 

76 

■96 

96 
96 



Finding 3A 
Finding 3B 
Finding 30 



Yes 


4) 


17 


13 


9 


24 


100% 




42 


44 


45. 


40 


75% 




69 


81 


91 


52 


50% 




77 


81 


100 


64 


25% 




79 


81 


100 


68 


0% 




79 


81- 


100 


68 


No Response 


100 


100 


100 


100 


Yes 


5) 


10 


125 


9 


8 


100% 




29 


25 


27. 




75% 




62 


75 


36. 


^64 


50% 




79 


88 


45. 


88 


25% 




81 


oo 


45 


92 


0% 




87 


94 


45 


100 


No Response 


100 


100 


100 


100 


Yes 


6) 


17 


19 


18 


16 


100% 




30 


25 


36 


32 


75% 




70 


69 


91. 


— 64 


50% 




80 


75 


91— 80 


25% 




86 


75 


100 


88 


0% 




88 


75 


100 


92 


No Response 


100 


100 


100 


100 


Yes 


7) 


38 


19 


73. 


36 


No 




100 


100 


100 


100 


Yes 


8) 


8 


5.5 


9 


8 


100% 




23 


18.5 


18 


28 


75% 




62 


68.5 


46. 


64 


50% 




81 


81.5 


82 


— 80 


25% 




96 


94.5 


91 


100 


0% 




98 


94.5 


100 


100 


No Response 


100 


100 


100 


100 



Finding 4A 
Finding 4B 



Finding 5A 
Finding 5B 
Finding 5C" 



Finding 6A 
Finding 6B^ 
Finding 6C 



Finding 7 



Finding 8A 
Finding 8B 



1-1 

01 

> OQ 
C CO 

(U C H 
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O O PQ CO 



p< 



,01 



p <.io 
p<^.ii 



P < .05 



P<.01 P<.05 
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DISPUY 1 

COMPUTING THE SIGNIFICANCE OF A DIFFERENCE IN PERCENTAGE 

PSN ■= Number of P.S. (read from Table 1, question 0). 

P.S.% = 7. (read from Table 3 or 4 for each question), 

CIN = Number of C.I. (read from Table 1, question 0). 

C.I.% = 7o (read from Table 3 or 4 for each question). 



DIFFERENCE 
IN 

PERCENTAGE 



Standard Deviation 

of the = (5^ diff 

Difference 



(P.S.%) a007. - P.S. 7.) 
PSN 



(C.I .7.) (1007. - C.I. 7.) 
CIN 



Difference in Percentage 
Difference 



I z -chart" 



If Z is greater than: 


Then the level of confidence is: 




7. 


Proportion 


Chances 


1.29 


207. 


.20 


1 in 5 


1.64 


107o 


.10 


1 in 10 


1.96 


57. 


.05 


1 in 20 


2.57 


17. 


.01 


1 in 100 


3.27 


.17. 


.001 


1 in 1000 
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DISPLAY 2 

Tabulating Table for Significance Testing 
(Slide Rule Accus'acy) 



Raw Content Step 



Item being tested 
statistically 


o 


4B 


5B 


5C 


6B 


7 






TOTAL Number in Group I 


© 


11 


11 


11 


11 


11 






% of Group 1 Giving 
Response "R" to Question "Q" 


© 


91 


36 


45 


91 


73 






100% minus Step 2 


© 


9 


64 


55 


9 


27 






Rounded Off 

Product of Step 1 Multiplied 
bv Step 2 


© 


820 


2300 


2480 


820 


1970 






Step 4 Divided by Step 1 


® 


TT 


2300 
11 


2480 
11 


820 
11 


1970 
11 







Answer to Step 5 


® 


75 


210 


225 


74.5 


178 






TOTAL Number in Group II 


© 


25 


25 


25 


25 


25 






% of Group II Giving 
Response "R" to Question "Q". 


® 


52 


64 


88 


64 


36 






100% minus Step 8 


® 


48 


36 


12 


36 


64 






Rounded Off 

Product of Step 8 Multiplied 
by Step 9 


@ 


2500 


2300 


1055 


2300 


2300 






Step 10 Divided by Step 7 


® 


25 


25 


25 


25 


25 






Answer to Step 11 


© 


100 


92 


42 


92 


92 






Sum of Step 6 and Step 12 


® 


175 


302 


267 


166.5 


270 






Square Root of Step 13 




13.2 ' 


17.4 


16.3 


12.9 


16.4 






Difference Between Step 12 
and Step 8 




91 - 48 
=39 


64 - 36 
=28 


88 - 45 
=43 


91 - 64 
=27 


73 - 36 
=37 






Step 15 Divided by Step 14 




39 
13.2 


28 
17.4 


43 
16.3 


27 
12.9 


37 
16.4 






Answer to Step 16 


© 


2.95 


1.63 


2.64 


2.09 


2.26 






nVV 1 of Probability 
Z-Chart 


© 


p <.oi 


P <.ll 
p^.lO 


.p <.01 


P <.05 


p<.05 

p>-oi 
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DISPLAY 3 

Percentage Equivalents 
On a Base of 11 



1/11 = 97o 

2/11 = 187o 

3/11 = 277o 

4/11 = 367o 

5/11 = 467o 

6/11 = 557o 

7/11 = 647o 

8/11 = 737o. 

9/11 = 827o 

10/11 = 9l7o 

11/11 = 1007o 



56 



DISPLAY 4 



Percentage Equivalents 
On a Base of 16 



1/16 = 6% 
2/16 = 13% 
3/16 = 19% 
4/16 = 25% 
5/16 = 31% 

6/16 = 38% 
7/16 = 44% 
8/16 = 50% 
9/16 = 56% 
10/16 = 63% 

11/16 = 69% 
12/16 = 75% 
13/16 = 81% 
14/16 = 88% 
15/16 = 94% 
16/16 = 100% 
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DISPLAY 5 



Percentage Equivalents 
On a Base of 52 



1/52 = 2% 
2/52 = 4% 
3/52 = 6% 
4/52 = 8% 
5/52 = 10% 

6/52 = 12% 
7/52 = 13% 
8/52 = 15% 
9/52 = 17% 
10/52 = 19% 

11/52 = 21% 
12/52 = 23% 
13/52 = 25% 
14/52 = 27% 
15/52 = 29% 

16/52 = 31% 
17/52 = 33% 
18/52 = 35% 
19/52 = 37% 
20/52 = 38% 

21/52 = 40% 
22/52 = 42% 
23/52 = 44% 
24/52 = 46% 
25/52 = 48% 

26/52 = 50% 

27/52 = 52% 

28/52 = 54% 

29/52 = 56% 

30/52 = 58% 



31/52 = 60% 
32/52 = 61% 
33/52 = 63% 
34/52 = 65% 
35/52 = 67% 

36/52 = 69% 
37/52 = 71% 
38/52 = 73% 
39/62 ^= 75% 
40/52 = 77% 

41/52 = 79% 
42/52 = 80% 
43/52 = 82% 
44/52 = 84% 
45/52 = 86% 

46/52 = 88% 

47/52 = 90% 

48/52 = 92% 

49/52 = 94% 

50/52 = 96% 

51/52 = 98% 
52/52 = 100% 



58 



COMMENTS ON ITEM 1 



NO IDENTIFICAT ION 



PUBLIC SCHOOL 



CORRECTIONAL INSTITUTION 



Exchanged views , problems, 
Ideas^ course content, 
equivalent information. 

It is an opportunity to 
receive an insight of the 
public schools concept of 
vocational education. 

Exchanged ideas from each 
other and help. 

Exchange of ideas methodology. 

Exchange of ideas . 

It provided an excellent 
opportunity to exchange ideas 
and to discuss various train- 
ing methods as well as lesson 
or ganization. 

Getting to know each other and 
the free flow of mutual 
information. 

We got to know that we all 
pretty much have the same 
problems • 

A better understand of each 
other's problems. 

Exchange of ideas. 

We more fully understand our 
mutual problems that we did 
no know about before. 

Exchange of ideas « 

By comparing shops and 
trade information. 

Interchange of ideas. 

Have a better understandings 
of their operation. Getting^^ 
to know other teachers. 



Understanding exchange of 
views. 

Exchange of views. 

Exchange of ideas, materials^ 
address's for free 
information and materials. 

Conmunication and exchange 
of ideas. 

Interaction between both 
D.C.S. and public school 
instructors . 

We teach similar curricultmis 
and will exchange teaching 
aides and A-V materials. 

Exchange of ideas. 

Idea exchange. Development 
of objectives. 

The CI teachers seem well 
informed and interested. 
We have much to offer them 
in areas of f acuity-equip. - 
behavioral objectives . 

Mutual exchange of ideas 
and interesting 
experiences . 



By interaction with peers 
and also by finding out 
shop layouts . 

Resolving mutual problems. 
Feeling of confidence that 
we are following similar 
teaching patterns. 

Exchange of ideas. 

Some of the problems are 
the same. 

Exchange of home and phone 
address. Discussion of 
mutual problems. 

It was a chance to meet with 
other related trade instructors 
and discuss common problems. 

Exchange of ideas and info, 
of recent issue. 

By exchange of ideas-views 
similar problems. 

Exchange of concepts & ideas 
were of great value. 

Interaction of ideas, methods 
and techniques . 

Interaction in the form of 
exchange of methodology and 
materials . 

Same basic problems. 

Exchange of ideas and a hand- 
out w/names & addresses of 
those in attendance. 

Exchange of ideas and teaching 
concepts . 

i 

Understand each other problems. 



(15 comments) 



ERIC 



(10 comments) 
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(Continued on next page) 
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COM^NT§^^N ITEM 1 




NO IDENTIFICATION 


PUBLIC SCHOOL 


, CORRECTIONAL .INSTITUTION 






Exchanging of ideas and 
forming a beginners 
test. 






Better understanding 
between Instructors. 






Idea exchange. 






Exchange of teaching Ideas 
and operation of shops. 






Exchange of Information. 



(20 comments) 



(Grand total 45) 
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COMMENTS ON ITEM 3 
(75% or above) 



56 



NO IDENTIFICATION 



PUBLIC SCHOOL 



CORRECTIONAL INSTITUTION 



A better means of beginning 
£c end results. 

Help in teaching new class 
of students. 

Developing a course outline 
for Heating that we did 
not have. 

Perhaps it will enable 
myself, being a new voc. inst 
to better organize my lesson 
plans . 

Gave us a better teaching plan 

Better knowledge. 

This module will become a 
lesson plan for my program. 

Test student entrance ability. 



(8 comments) 



100% as of Sept, 75 - in the 
process of establishing 
completed modules now for 
75-75 school years. 

Tool control, discipline 
measurement. 

Time saver. 

Course outlined course 
literature exchange thru 
mail . 

To develop modular lesson to 
help the changing population 

As a start to the entire 
course. 

Help to individualize 
instruction. 



(7 comments) 



Incomplete as it now exists, 

I didn't need another test. 
I have adequate ones to fit 
my situation. 

Course outlines cope with 
changing population. 



(50% or below) 

It is a worklrig module that 
will be used for that 
subject . 



(3 comments) 



(1 comment) 



Because of previous exposure 
and interest in module 
teaching. 

Developing module lesson 
and outline to cope with 
quick changing student 
population. 

By developing an outline. 

Develop outline and literature 
through instruction. 

Can be used on an 
individual basis - expect 
to carry on. 

New shop - new instructor - 
gives me a sense of direction. 

Better concept of what direction 
of inst. approaches to use. 

It completes one module now 

for expansion on the other, fC ont 



By covering the ref . aspect 
of major/minor application. 

More definite course of 
study. 

I am involved in diesel 
Engines . 



(3 comments) 
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COMMENTS ON ITEM 3 
(75% or above) 
(Con't) 




NO THFNTT'PTnATTn'W 


PTTRT TP ^rHOOT 


1 CORRECTIONAL INSTITUTION 






On pretesting men. 






In curriculum design. 






Standard between faculty. 






To move in some future 
direction. 






It will produce better 
students in knowing 
what level in starting. 



(Continued from 75% or above) 
(13 comments) 



(Grand total 35) 



I 
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COMMENTS ON ITEM 4 
(75% or above) 



NO IDENTIFICATION 



Motivation course content. 

Security - rotational 
problems - inconsisency in 
assigning students. 

Being a correctional inst. 
myself, I don't feel I can 
honestly answer this question. 

More, realistic teaching 
situations greater use of 
existing methodology. 

Better methods of classification 

Enrollment - security. 

Tools and equipment. 

Zookeeping - types of men, 
inst. problems. 

Student selection & motivation. 



PUBLIC SCHOOL 



Cooperation - interaction. 

High student turn out - 
lack of time. 

Motivation - the changing 
enrollment of the class. 

■ 

New students entering anytime 
Priority of security over 
school. 

Restrictions . 

Ideas of motivation similar 
to public but much more so. 

This sounds very much like 
ours. 

Student turnover and lack 
of equipment. 

(8 comments) 



CORRECTIONAL INSTITUTION 



Because of previous . 
exposure & interest in 
module teaching.. 

Student problems. 

Administration problems,, 
security situation. 

Placement. 

Security assignment of 
students, time out of 
classroom. 

Individual instruction? 

Student assignment & 
interest* 

Disciplining. 

Better student placement. 

(8 comments) 



(8 comments) 



(507o or below) 
Student motivations. 



Type of manpower. 
Levels of learning. 



(1 comment) 



(2 comments) 



(Grand total 27) 
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COMMENTS ON ITEM 5 
(75% or above) 



5? 



NO IDENTIFICATION 


PUBLIC SCHOOL 


CORRECTIONAL INSTITn:iON 


By exchanging course content 
and information. 

The benefits of entry tests 
and posttests. 

It enables me to receive a 
better scope of materials 
available and their uses. 

Hope so. 

How to motivate the inmate 
morp f> ff PC tivplv 

By more detailed information. 
Interchange of ideas. 


Cooperation interaction. 

Similar problems. 

Deyelop course outline 
interaction between group' 
started. 

- 


Type of paperwork, etc. 

Limits involved with different 
level students. 

A definite uniform 
curriculum is needed. 

Understanding performance 
objectives. 

Because of type of student 
and other basic problems. 

By simplification. 

Mentality we are dealing with. 

An understanding. 


(7 comments) 


(3 comments) 


(8 comments) 




(50% or below) 




Interaction. 
Interaction. 

I 
» 

« 


Limited time to. deyelop 
meaningful measurement 
devices . 


New material. 
Specific techniques. . 
Interaction. 

We all attended the same 
programs and courses. 

Mod erni za ti on . 

Positive approach. 

New and more methods of 
obtaining materials. 


(2 comments) 


(1 comment) ' • 


(7 comments) 
(Grand total 28) 
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COMMENTS ON ITEM 6 
(75% or above) 



NO IDENIIFICATION 


PUBLIC SCHOOL 


CORRECTIONAL INSTITUTION 


Taking over class while gone 
during visitation. 


Various areas ^ were touched 


Live work - money - 
supplies. 


Visitation. 
Protocol 


Bureaucracy and red tape 
hang ups . Adminis tra tion 
reluctance to face problems. 


Administration security. 
Security. 


Administrative duties. 


Security possibly. 


Visitations. 


Administrative security . 
problems. 

Reading ability. We all have 
had similar problems. 
* 


Protocol for visits. 

Visitations . 

Learning, teaching, 
motivation. 


Parts - correspondence - 
need to know. 

Considerable number 
visitation too numerous 
to mention. 


• 


Poorly organized, more time 


Lack of standardization 
of courses « 




Bureaucratic protocol. 


Security. 


(6 comments) 




comments) 




(50% or below) 




Meeting, correspondance, 
chain of command, procedures. 


Administration problems. 


Visitations. , - 


Classification of students. 




Security administration. 


Security and motivation. 




Attendance, motivation, and 
student learning ability. 

Learning motivation, etc. 

Problems with below 
average students. 


(3 coramenfis) 


(1 comment) 


(5 comments) 
(Grand total 31) 

• * 
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COMMENTS ON ITEM 7 
YES 



NO IDENTIFICATION 


PUBLIC SCHOOL 


CORRECTIONAL INSTITUTION 


Developed curriculum in 
medical electronics using 
behavioral objectives. 

My own course of study was 
written that way. 


In process of establishing 
modules for my program. 

Workshop inservice. 

Other workshops. 


Private business. 

College curriculum. 

Oswego curriculum 
development. 


Apprentice programs 


Workshops and modules on 
performance objectives. 


I was on their advisory 
group. 




Developed. 
Inservice workshops. 
At own facility. 


BOCES occupational 
analysis VED? 

UT courses. 

School. 

In teaching with Ford 
Motor Co. 


(3 comments) 


(1 coTnmPnts'i 


Divisions job 
sequences . 

(9 comments) 




NO 




I have heard of them. 




Nev/ in education field. 


(1 comment) 


(0 comments) 


(1 comment) 
(Grand total 21) 
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COMMENTS ON ITEM 8 
(75% or above) 



NO IDENTIFICATION 



Helped me understand 

the concept* 4 little better. 

A better means of measuring 
your accomplishments. 

By mutual cooperation. 

By showing me the value 
and proper application. 

New ideas. 

Hand out materials. Ideas from 
others on what and how. 

Give me basic concept 
(esp. Jun B) 

I now have an understanding of 
module construction. 

(8 comments) 



PUBLIC SCHOOL 



Advantage of module for 
random enrollment and 
specific testing. 

Additional resources.. 



(2 comments) 



CORRECTIONAL INSTITUTION 



Clarity terms and better 
understanding of a module. 

By actual participation. 

Confirmed my belief of 
the need of this system 
because of the type and 
turnover of students 
under open enrollment. 

Better outlines. 

Have better understanding 
of how it*s prepared. 



(5 comments) 



We lack a valid skill list 
for radio-TV technology. 

Better understand of what 
was wanted. 

Reinforced. 

How to present new 
procedures in handling 
problem cases. 

Basic definition outline. 



(50% or below) 
Pretest and entry. 

Basic understanding of 
format. 

Module development explained. 

Define modular approach. 

Brought out ideas not 
thought pertinent before. 

Some insight on top of 
present knowledge. 



(5 coimnents) 



(6 comments) 
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More understanding of how 
a modular is set up. 

Outline of modules and 
definition of modules. 

Basic outline. 

The vehicle to establish 
same. 

Definition, design, 
concepts . 

Because of the turnover 
in the students . 

Knew nothing about modules 
tefore* 

To simplify our beginners 
under s tand ing . 

By teaching students at 
many different levels. 

(9 comments) 

(Grand total 35) 
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COMMEOTS ON THE BACK OF THE EVALUATION PAGE 



NO IDENTIFICATION 



This has been a helpful and meaningful experience. Will have a great 
effect in teaching in the future. Interagency cooperation will be 
fruitful for all concerned. 

In general, the prdgram provided a valuable exchange of information 
between intragroup members. I don't believe we completed the objectives 
as outlined in the memo provided the participants. 

No enough time for informal discussion. We're just geting to know each 

other and are comfortable and it's over. More time should have been 

spent exchanging successful teaching methods — programs — audio-visual, etc. 

We should have more of these informative sessions so that we can exchange 
ideas, problems, equipment, training methods, etc. 

If prior to coming, the instructors were asked to get down on paper, ideas 
or modules now in use. Some of the modules developed by one group could 
be of use by other trades, (e.g., a measurement module developed by 
Plumbing could be used by auto body, etc. These modules could be used 
picked over for ideas and directions.) One of the pre or post test could 
be used by other trades. 

The interaction emphasis was of definite help to all involved. Mailing 
lists were obtained to exchange information. 

Good meeting. Suggest that coffee be available. Also directory of 
participants could easily be ready by end of session, (especially if 
information form were filled out at region)* 

I hope this will not be the end of this cooperative effort. Please follow 
up on a good thing that we have started. 
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COMMENTS ON THE BACK OF THE EVALUATION FORM 
PUBLIC SCHOOL 



About the only comment I could possibly make is now that a communicative 
stream has started, keep it moving. If another institute like this should 
be in the plans in the near future, I would recommend that participating 
cjo-op teachers.be advised so that we have an opportunity to better prepare 
P.S. I enjoyed it and it was productive. 

In my opinion, another meeting or workshop is of the utmost importance. 
Perhaps, of longer duration. A more specific outline of objectives (to be 
given by the leaders) has to be given. Vague approaches only lead to 
useless expense of time. We all got a feeling of mutual understanding o^ 
each others problems. Perhaps, as much was accomplished "after hours" as 
during hours. 

We intend to keep mailing to each other literature. Plus visit each 
others shops. 

I feel that a working relation between the D.C.S and myself will continue 
and hopefully inspire the educational systems both in the D.C.S. and the 
public school systems . 

If one of the objectives was to write modules in performance objective^" 
then the workshop should have had someone with experience in writing these 
occupational education (psychomotor domain). This would have trained all 
participants in writing modules not confusion. 

This workshop should have followup meetings and extended cooperation 
between corr-and ITT persons. Specific problems which should be dealb 
with in both areas: student entry level and screening. 

More time should be given to ideas of teachers. Most of the available 
study time was used to develop written work to be handed out. In some 
cases, this did not product the best results possible. Too much time 
available used on introductory remarks and presentations not exactly 
meaningful to situation. Hard to use or adapt acddeimic situation to 
vocational. 
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COMMENTS ON THE BACK OF THE EVALUATION PAGE 
CORRECTIONAL INSTITUTION 



The meeting was very constructivey Created » A feeling of resolving 
mutual problems in a well conducted manner. Thank you. 

I would like to see more meetings of the type --longer periods. 

An excellent example of professionals comparing cooperating and 
developing a usable product. More followup is needed and visitation 
seems an excellent method for professional interaction. More of this 
interaction between education are needed. ^ 

Encountered in this seminar, was a feeling of cooperation, need for more 
exposure to different approaches in presenting materials and general 
exchange of resources a very meaningful experietlce. 

Because of the number of students added at any time during the school 
calendar I have been using a form of this type of program. I have used 
a type of test that tells me, something about this general knowledge. 

I think the meeting was very beneficial and future meetings of this 
type should be considered. 

I feel the need for a committee to develop standard module to be 
adopted by all faculty. 

Development of the module concept should be presented more fully be 
presenting actual module teaching demonstration in a how to situation. 
Possibly a course in module teaching concept. 

It is a great advantage for the group to exchange ideas and better ways 
to teach by other skilled tradesman ideas. We should meet at least every 
six months to better serve our curriculum outlines. 



ERLC 
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PART III 



HOtf TO APPLY THIS EVALUATION APPROACH TO 
DIFFERENT SITUATIONS 

Part I has stressed the difference between an evaluation suimnary and 
a complete evaluation report. 

Part II has shown how to develop evaluation summaries and complete 
evaluation reports. 

Part III shows how to apply this evaluation approach to a wide variety 
of different situations. 
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SITUATION A 
CHARACTERISTICS OF SITUATION A 



A large group meeting is conducted. 

Each participant is asked to fill out an 
evaluation form. 

The participants are grouped into a minimum 
of two subgroups. 

Tabulation is made by either an external or 
by the institute director. 



An example of Situation A is found in Part I and Part II of this 
document. 



TYPICAL VARIATIONS: 



The external evaluator tabulates the data. 
The institute director tabulates the data. 
Both the institute director and the external 

evaluator tabulates the same data and 

compare conclusions. 
Either the external evaluator or the institute 
. director take a separate sampling. 
When the group is large, only a 6 percentage of 

each subgroup fills out an evaluation form. 
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SITUATION B 



CHARACTERISTICS OF SITUATION A 



The same as Situation A except that the 
institute director also fills out an evaluation 
form. 

The external evaluator receives the data 
from the participant evaluators. 

The external evaluator receives the data 
from the institute director. 

Before looking at the evaluation form filled 
'out by the institute director, the external evaluator 
goes through all the processes spelled out in Part I 
and Part II. 

With this done, the external evaluator compares 
the responses of the group, subgroups, and institute 
director . 

Agreements are pinpointed. Disagreements are 
pinpointed and discussed for resolution. 

This type of evaluation helps the institute 
director realize how well the group consensus has 
been ascertained through informal feedback. 



TYPICAL VARIATIONS: 



The institute director and the institute staff 
fill out the evaluation forms. 

When the number of participants is small, subgroups 
are avoided and the institute director is compared 
directly to the group total. 

A particular power of variation B is the fact that: 
Sometimes , the institute director has an awareness 
of the generalized group reaction without being able 
to break this reaction down into significant subgroups. 
Sometimes, the institute director is aware of the 
reaction of only one of the subgroups and not in 
contact with reactions of other subgroups or of the 
total group. 
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PERSPECTIVES 
FOR EVALUATING THE INSTITUTES 
BETWEEN CORRECTIONAL TEACHERS AND VOCATIONAL TEACHERS 



How many readers of this page can tell exactly what is meant by a 
Flexible device 

for consolidation of quantified but amorphous forces 

into a linear form 

and into vectors selected from 

a poly-directional array? 

The way the above question is put complicates any attempt to arrive 
at an answer, this definition of a TOW ROPE is indeed very technical, very 
accurate, and easily quantified • Unfortunately, it is not the type' of a 
definition that helps correctional teachers and occupational teachers pull 
together • 

In setting up this evaluation design, consideration has been given to 
a number of good statistical techniques, such as co-variance, multi-regressi 
and bifurcation. The final choice of evaluation design has been based upon 
the desire to create a meaningful evaluation, 

A MEANINGFUL EVALUATION 

A meaningful evaluation assumes that both correctional teachers and 
occupational teachers can work together and will continue to have joint 
institutes in the future. These joint institutes will undoubtedly follow the 
patterns and presidence set by the first institute. 
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It is important that the evaluation design for the first institute 
prepares for future institutes. 

In other words, this evaluation is looking for both good and bad 
results of the initial collaboration effort. The institute planners will 
improve the bad results. The instituteplanners will reinforce and utilize 
the good results. 

This is why the following evaluation form stresses as the number one 
priority the determination of WHAT WENT RIGHT and WHAT WENT WRONG with the 
initial institute. 
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